<!DOCTYPE html>
<html>
<head>
<title>周志华《机器学习》课后习题解答系列（七）：Ch6.3 - SVM对比实验</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
/* GitHub stylesheet for MarkdownPad (http://markdownpad.com) */
/* Author: Nicolas Hery - http://nicolashery.com */
/* Version: b13fe65ca28d2e568c6ed5d7f06581183df8f2ff */
/* Source: https://github.com/nicolahery/markdownpad-github */

/* RESET
=============================================================================*/

html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
  margin: 0;
  padding: 0;
  border: 0;
}

/* BODY
=============================================================================*/

body {
  font-family: Helvetica, arial, freesans, clean, sans-serif;
  font-size: 14px;
  line-height: 1.6;
  color: #333;
  background-color: #fff;
  padding: 20px;
  max-width: 960px;
  margin: 0 auto;
}

body>*:first-child {
  margin-top: 0 !important;
}

body>*:last-child {
  margin-bottom: 0 !important;
}

/* BLOCKS
=============================================================================*/

p, blockquote, ul, ol, dl, table, pre {
  margin: 15px 0;
}

/* HEADERS
=============================================================================*/

h1, h2, h3, h4, h5, h6 {
  margin: 20px 0 10px;
  padding: 0;
  font-weight: bold;
  -webkit-font-smoothing: antialiased;
}

h1 tt, h1 code, h2 tt, h2 code, h3 tt, h3 code, h4 tt, h4 code, h5 tt, h5 code, h6 tt, h6 code {
  font-size: inherit;
}

h1 {
  font-size: 28px;
  color: #000;
}

h2 {
  font-size: 24px;
  border-bottom: 1px solid #ccc;
  color: #000;
}

h3 {
  font-size: 18px;
}

h4 {
  font-size: 16px;
}

h5 {
  font-size: 14px;
}

h6 {
  color: #777;
  font-size: 14px;
}

body>h2:first-child, body>h1:first-child, body>h1:first-child+h2, body>h3:first-child, body>h4:first-child, body>h5:first-child, body>h6:first-child {
  margin-top: 0;
  padding-top: 0;
}

a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
  margin-top: 0;
  padding-top: 0;
}

h1+p, h2+p, h3+p, h4+p, h5+p, h6+p {
  margin-top: 10px;
}

/* LINKS
=============================================================================*/

a {
  color: #4183C4;
  text-decoration: none;
}

a:hover {
  text-decoration: underline;
}

/* LISTS
=============================================================================*/

ul, ol {
  padding-left: 30px;
}

ul li > :first-child, 
ol li > :first-child, 
ul li ul:first-of-type, 
ol li ol:first-of-type, 
ul li ol:first-of-type, 
ol li ul:first-of-type {
  margin-top: 0px;
}

ul ul, ul ol, ol ol, ol ul {
  margin-bottom: 0;
}

dl {
  padding: 0;
}

dl dt {
  font-size: 14px;
  font-weight: bold;
  font-style: italic;
  padding: 0;
  margin: 15px 0 5px;
}

dl dt:first-child {
  padding: 0;
}

dl dt>:first-child {
  margin-top: 0px;
}

dl dt>:last-child {
  margin-bottom: 0px;
}

dl dd {
  margin: 0 0 15px;
  padding: 0 15px;
}

dl dd>:first-child {
  margin-top: 0px;
}

dl dd>:last-child {
  margin-bottom: 0px;
}

/* CODE
=============================================================================*/

pre, code, tt {
  font-size: 12px;
  font-family: Consolas, "Liberation Mono", Courier, monospace;
}

code, tt {
  margin: 0 0px;
  padding: 0px 0px;
  white-space: nowrap;
  border: 1px solid #eaeaea;
  background-color: #f8f8f8;
  border-radius: 3px;
}

pre>code {
  margin: 0;
  padding: 0;
  white-space: pre;
  border: none;
  background: transparent;
}

pre {
  background-color: #f8f8f8;
  border: 1px solid #ccc;
  font-size: 13px;
  line-height: 19px;
  overflow: auto;
  padding: 6px 10px;
  border-radius: 3px;
}

pre code, pre tt {
  background-color: transparent;
  border: none;
}

kbd {
    -moz-border-bottom-colors: none;
    -moz-border-left-colors: none;
    -moz-border-right-colors: none;
    -moz-border-top-colors: none;
    background-color: #DDDDDD;
    background-image: linear-gradient(#F1F1F1, #DDDDDD);
    background-repeat: repeat-x;
    border-color: #DDDDDD #CCCCCC #CCCCCC #DDDDDD;
    border-image: none;
    border-radius: 2px 2px 2px 2px;
    border-style: solid;
    border-width: 1px;
    font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
    line-height: 10px;
    padding: 1px 4px;
}

/* QUOTES
=============================================================================*/

blockquote {
  border-left: 4px solid #DDD;
  padding: 0 15px;
  color: #777;
}

blockquote>:first-child {
  margin-top: 0px;
}

blockquote>:last-child {
  margin-bottom: 0px;
}

/* HORIZONTAL RULES
=============================================================================*/

hr {
  clear: both;
  margin: 15px 0;
  height: 0px;
  overflow: hidden;
  border: none;
  background: transparent;
  border-bottom: 4px solid #ddd;
  padding: 0;
}

/* TABLES
=============================================================================*/

table th {
  font-weight: bold;
}

table th, table td {
  border: 1px solid #ccc;
  padding: 6px 13px;
}

table tr {
  border-top: 1px solid #ccc;
  background-color: #fff;
}

table tr:nth-child(2n) {
  background-color: #f8f8f8;
}

/* IMAGES
=============================================================================*/

img {
  max-width: 100%
}
</style>
</head>
<body>
<p>查看相关答案和源代码，欢迎访问我的Github：<a href="https://github.com/PY131/Machine-Learning_ZhouZhihua">PY131/Machine-Learning_ZhouZhihua</a>.</p>
<h2>6.3 SVM对比实验</h2>
<blockquote>
<p><img src="Ch6/6.3.png" /></p>
</blockquote>
<p>本题实验基于python，各种算法的实现基于的开源工具包和源码对应如下：</p>
<ul>
<li>SVM -&gt; sklearn</li>
<li>BP -&gt; pybrain</li>
<li>C4.5 -&gt; 来自<a href="https://github.com/ryanmadden/decision-tree">GitHub/ryanmadden/decision-tree</a></li>
</ul>
<p><a href="https://github.com/PY131/Machine-Learning_ZhouZhihua/tree/master/ch6_support_vector_machine/6.3_SVM_compare">这里查看本实验完整代码</a></p>
<p>这里我们选择UCI数据集 <a href="http://archive.ics.uci.edu/ml/datasets/Breast+Cancer">Breast Cancer Data Set </a> 进行分类实验，该数据集已经集成在sklearn中，可以直接采用<code>sklearn.datasets.load_breast_cancer</code>获取。</p>
<h3>数据预处理分析</h3>
<p>关于该数据集的相关信息参考sklearn官网<a href="http://scikit-learn.org/dev/modules/generated/sklearn.datasets.load_breast_cancer.html#sklearn.datasets.load_breast_cancer">sklearn.datasets.load_breast_cancer</a>，关键信息如下：</p>
<pre><code>类别：'malignant(恶性)' 'benign(良性)'，共两类；
样本规模：[n_sample,n_feature] = [569, 30],；
         正负样本数为 212(M),357(B)；
         特征数值特性：实数，正值；
</code></pre>

<p>加载数据，选择前两个特征可视化出散点图如下所示：</p>
<p><img src="Ch6/6.3.scatter.png" /></p>
<p>上图中，绿色为良性(肿瘤)，红色对应恶性，于是可直观看出，体积越大恶性概率越高。</p>
<p>经过预分析，我认为该数据集可以直接用于SVM，BPnet，C4.5的模型学习。</p>
<h3>模型训练与测试</h3>
<p>首先将数据集随机划分为相等了两部分，一部分作训练，一部分作测试。然后采用三种模型来拟合训练集，并在测试集上进行分类预测。在训练SVM和BP网络时，需对数据进行归一化处理，这里采用<code>sklearn.preprocessing.normalize</code>实现。</p>
<p>划分数据集与归一化程序如下示：</p>
<pre><code>```python
from sklearn import preprocessing
from sklearn.cross_validation import train_test_split

normalized_X = preprocessing.normalize(X)
X_train, X_test, y_train, y_test = train_test_split(normalized_X, y, test_size=0.5, random_state=0)
```
</code></pre>

<p>下面是三种模型的训练和测试的程序和结果示意：</p>
<ol>
<li>
<p>SVM - 基于sklearn，分别采用线性核和高斯核进行实验。</p>
<pre><code>```python
clf = svm.SVC(C=C, kernel=kernel) kernel
# train
clf.fit(X_train, y_train)
# testing 
y_pred = clf.predict(X_test)
```
</code></pre>

<p>得出分类精确度如下：</p>
<pre><code>max accuracy of linear kernel SVM: 0.937
max accuracy of rbf kernel SVM: 0.933
</code></pre>

</li>
<li>
<p>BP net - 基于pybrain，采用“标准BP算法+Softmax输出层激活函数”构建二分类器。</p>
<pre><code>```python
from pybrain.tools.shortcuts     import buildNetwork
from pybrain.structure.modules   import SoftmaxLayer
from pybrain.supervised.trainers import BackpropTrainer

n_hidden = 600
bp_nn = buildNetwork(trndata.indim, n_hidden, trndata.outdim, outclass=SoftmaxLayer)
trainer = BackpropTrainer(bp_nn, 
                          dataset=trndata,
                          verbose=True,
                          momentum=0.2,
                          learningrate=0.0002)
err_train, err_valid = trainer.trainUntilConvergence(maxEpochs=1000)
```
</code></pre>

<p>得出测试集分类精确度如下：</p>
<pre><code>epoch: 1001  test error:  6.69%
</code></pre>

</li>
<li>
<p>C4.5，参考<a href="https://github.com/ryanmadden/decision-tree">ryanmadden/decision-tree - GitHub</a>。</p>
<p><a href="https://github.com/PY131/Machine-Learning_ZhouZhihua/blob/master/ch6_support_vector_machine/6.3_SVM_compare/c45_clf.py">点击查看C4.5实验源码</a></p>
<p>得出测试集分类精确度如下：</p>
<pre><code>accuracy of C4.5 tree: 0.909
</code></pre>

</li>
</ol>
<h3>模型对比</h3>
<ul>
<li>
<p>资源消耗：BP神经网络训练消耗计算机资源最大，SVM也比较耗资源，C4.5决策树消耗资源最小，训练速度极快。</p>
</li>
<li>
<p>参数调节：BP网络的精度受模型结构和参数的设定影响较大，需要耐心的调参。不同核函数下的SVM精度与数据对象特性息息相关，实践中也是结合参数（如惩罚系数）要不断调试的，决策树-C4.5算法则相对固定一些。</p>
</li>
<li>
<p>模型精度：只要模型与参数设置得当，经过足够的训练甚至交叉验证等，三种模型均可表现出不错的精度。</p>
</li>
</ul>
<h3>参考</h3>
<p>下面列出本实验涉及到的一些重要的参考资料：</p>
<ul>
<li><a href="http://pybrain.org/docs/tutorial/fnn.html">Pybrain官网样例 - Classification with Feed-Forward Neural Networks</a>；</li>
<li><a href="https://github.com/ryanmadden/decision-tree">ryanmadden/decision-tree</a>：来自于GitHub的C4.5算法决策树分类器源码；</li>
</ul>

</body>
</html>
<!-- This document was created with MarkdownPad, the Markdown editor for Windows (http://markdownpad.com) -->
